Significant developments in mosaic filter technologies have paved the way to full motion video (FMV) mosaic-array multi-spectral cameras. The new class of multi- and hyper-spectral cameras opens broad possibilities of its utilization for military and industry purposes. Indeed, such cameras are able to classify materials as well as detect and track spectral signatures continuously in real time while simultaneously providing an operator the benefit of enhanced-discrimination-color video. Supporting these extensive capabilities requires significant back-end computational processing of the collected spectral data. In general, two processing streams are envisioned for mosaic array cameras. The first is spectral computation that provides essential spectral content analysis, e.g. detection or classification. The second is presentation of the video to an operator that can offer the best display of the content depending on the performed task, e.g. providing spatial resolution enhancement or color-coding of the spectral analysis. These processing streams can be executed in parallel or they can utilize each other's results. The spectral analysis algorithms have been developed extensively, however spatio-spectral processing of more than three equally-sampled spectral bands has been explored scarcely.
Mosaic-array cameras can have sensors with multiple short wave infrared (SWIR), or visible, bands of spectral resolution while requiring no custom optics. For example, sensors with at least 4, 9, 16, and 25 bands have been realized. The sensor can be composed of an array of pixel-sized filters fixed to a 2D staring focal plane array (FPA). In a 9-band sensor, the filter array can be composed of a repeating pattern of 3×3 unit cells, and each unit cell can contain a band-pass filters as well as short- or long-pass filters.
This type of sensor design can allow regaining some spatial resolution of the spectral bands when PSF size of the optics is order of the size of a single spectral pixel. Until now, the majority of the demosaicking processes has been developed for three-color Bayer color filter arrays; and, therefore, has provided spatial upsampling exclusively to RGB spectral bands with assumed strict relationship between them. Expansion of the techniques into multiple spectral bands has been attempted but with severely limiting assumptions. For example, in one technique, demosaicking is considered without spatial correlation of the spectral bands and is therefore substituted by standard image restoration using information from all available bands. In another technique, spatial correlation is utilized between the pixels, but critically relies on higher comparative resolution of one of the bands.
On the other hand, spatial correlation between adjacent pixels has been a driving force behind well-established multi-frame super-resolution. Additionally, super-resolution of hyperspectral images has been proposed. However, all hyperspectral super-resolution approaches in the prior art have to assume some type of spectral expansion and representation of hyperspectral data. For example, one hyperspectral super-resolution approach is Principal Component Analysis (PCA), but it is unattainable for multi-spectral data.
Accordingly, there remains a need in the art for a method for single-frame super-resolution based demosaicking of a multi-spectral single video frame.